In [33]:
plt.hist(p_diffs)
plt.axvline(obs_diff, c='red');
A/B tests are very commonly performed by data analysts and data scientists.
For this project, We will be working to understand the results of an A/B test run by an e-commerce website. Our goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
%matplotlib inline
#setting the seed
random.seed(42)
a. Read in the dataset from the ab_data.csv file and take a look at the top few rows here:
df = pd.read_csv('./ab_data.csv')
df.head()
| user_id | timestamp | group | landing_page | converted | |
|---|---|---|---|---|---|
| 0 | 851104 | 2017-01-21 22:11:48.556739 | control | old_page | 0 |
| 1 | 804228 | 2017-01-12 08:01:45.159739 | control | old_page | 0 |
| 2 | 661590 | 2017-01-11 16:55:06.154213 | treatment | new_page | 0 |
| 3 | 853541 | 2017-01-08 18:28:03.143765 | treatment | new_page | 0 |
| 4 | 864975 | 2017-01-21 01:52:26.210827 | control | old_page | 1 |
#Checking columns dtype and if there are null values.
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 294478 entries, 0 to 294477 Data columns (total 5 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 user_id 294478 non-null int64 1 timestamp 294478 non-null object 2 group 294478 non-null object 3 landing_page 294478 non-null object 4 converted 294478 non-null int64 dtypes: int64(2), object(3) memory usage: 11.2+ MB
# Converting timestamp column from object into datetime.
df['timestamp'] = pd.to_datetime(df['timestamp'])
# Calculating how many days this test took.
Days = df.timestamp.dt.date.max() - df.timestamp.dt.date.min()
print(f"This test was performed in {Days.days} days.")
This test was performed in 22 days.
b. Use the cell below to find the number of rows in the dataset.
print(f"This dataset has {df.shape[0]} rows and {df.shape[1]} columns.")
This dataset has 294478 rows and 5 columns.
What is the number of unique users in the dataset?
print("the number of unique users is : ",df.user_id.nunique())
the number of unique users is : 290584
c. The number of unique users in the dataset.
print("the proportion of users that converted is : ",np.round(df.converted.mean() * 100,2))
the proportion of users that converted is : 11.97
e. The number of times when the "group" is treatment but "landing_page" is not a new_page.
print("The number of times the new page and treatment don't line up is : ", df.query('landing_page != "new_page" & group == "treatment"').shape[0])
The number of times the new page and treatment don't line up is : 1965
f. Do any of the rows have missing values?
df.isnull().sum()
user_id 0 timestamp 0 group 0 landing_page 0 converted 0 dtype: int64
For the rows where treatment is not aligned with new_page or control is not aligned with old_page, we cannot be sure if this row truly received the new or old page , So I have to delete these rows and store it in new dataframe called df2.
# using pd.drop to delete the rows and storing the new dataframe in df2.
df2 = df.drop(df[((df['group'] == 'treatment') == (df['landing_page'] == 'new_page')) == False].index)
# Double Check all of the correct rows were removed - this should be 0
df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]
0
a. How many unique user_ids are in df2?
print(f"There are about {df2.user_id.nunique()} unique users. ")
There are about 290584 unique users.
b. There is one user_id repeated in df2. What is it?
print(f"The id of the repeated user is {df2.user_id.value_counts()[:1].index[0]} and it got repeated {df2.user_id.value_counts()[:1].values[0]} times.")
The id of the repeated user is 773192 and it got repeated 2 times.
c. Display the rows for the duplicate user_id?
df.query('user_id == 773192')
| user_id | timestamp | group | landing_page | converted | |
|---|---|---|---|---|---|
| 1899 | 773192 | 2017-01-09 05:37:58.781806 | treatment | new_page | 0 |
| 2893 | 773192 | 2017-01-14 02:55:59.590927 | treatment | new_page | 0 |
d. Remove one of the rows with a duplicate user_id, from the df2 dataframe.
#Using the index number to delete the row.
df2 = df2.drop(1899)
# Checking if it worked well !
df2.user_id.value_counts()[:1]
851104 1 Name: user_id, dtype: int64
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆
a. What is the probability of an individual converting regardless of the page they receive?
print("The probabitliy of and individual converting is : ",np.round(df2.converted.mean(),4))
The probabitliy of and individual converting is : 0.1196
b. Given that an individual was in the control group, what is the probability they converted?
prop = np.round(df2.groupby('group').converted.mean(),4)
print('Probability of Converting in Control group : ', prop[0])
Probability of Converting in Control group : 0.1204
c. Given that an individual was in the treatment group, what is the probability they converted?
print('Probability of Converting in Treatment group : ', prop[1])
Probability of Converting in Treatment group : 0.1188
#Calculating the difference between control group probability and treatment group probabitlity
obs_diff = prop[1] - prop[0]
print("The difference is : ",np.round(obs_diff,4))
The difference is : -0.0016
d. What is the probability that an individual received the new page?
print("The probability that an individual received the new page is : ", np.round((df2.landing_page == 'new_page').mean(),4))
The probability that an individual received the new page is : 0.5001
What is the probability that an individual received the old page?
print("The probability that an individual received the old page is : ", np.round((df2.landing_page == 'old_page').mean(),4))
The probability that an individual received the old page is : 0.4999
e. Consider your results from parts (a) through (d) above, and explain below whether the new treatment group users lead to more conversions.
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆
considering I need to make the decision just based on all the data provided. If I want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should my null and alternative hypotheses be? I will state my hypothesis in terms of words or in terms of $p_{old}$ and $p_{new}$, which are the converted rates for the old and new pages.
$$H_0: p_{new} \leq p_{old}$$
$$H_1: p_{new} > p_{old} $$
$$ \alpha = 0.05 $$
Assuming under the null hypothesis, $p_{new}$ and $p_{old}$ both have "true" success rates equal to the converted success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assuming they are equal to the convertedrate in ab_data.csv regardless of the page.
Using a sample size for each page equal to the ones in ab_data.csv.
Performing the sampling distribution for the difference in converted between the two pages over 10,000 iterations of calculating an estimate from the null.
a. What is the conversion rate for $p_{new}$ under the null hypothesis?
p_new = np.round(df2['converted'].mean(),4)
print("The convert rate for P_new is : ", p_new)
The convert rate for P_new is : 0.1196
b. What is the conversion rate for $p_{old}$ under the null hypothesis?
p_old = np.round(df2['converted'].mean(),4)
print("The convert rate for P_old is : ", p_old)
The convert rate for P_old is : 0.1196
c. What is $n_{new}$ , the number of individuals in the treatment group?
n_new = (df2['group'] == 'treatment').sum()
print(f"There are {n_new} individuals in the Treatment group.")
There are 145310 individuals in the Treatment group.
d. What is $n_{old}$ , the number of individuals in the control group?
n_old = (df2['group'] == 'control').sum()
print(f"There are {n_old} individuals in the Control group.")
There are 145274 individuals in the Control group.
e. Simulate Sample for the treatment Group
Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null hypothesis. Store these $n_{new}$ 1's and 0's in the new_page_converted numpy array.
new_page_converted = np.random.choice(2,size=n_new, p=[1 - p_new, p_new])
f. Simulate Sample for the control Group
Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null hypothesis.
Store these $n_{old}$ 1's and 0's in the old_page_converted numpy array.
old_page_converted = np.random.choice(2,size=n_old, p=[1 - p_old, p_old])
g. Find the difference in the "converted" probability $(p{'}_{new}$ - $p{'}_{old})$ for your simulated samples from the parts (e) and (f) above.
difference = new_page_converted.mean() - old_page_converted.mean()
print("The difference between P_new and P_old is : ", difference)
The difference between P_new and P_old is : 0.001291811826304487
h. Sampling distribution
Re-create new_page_converted and old_page_converted and find the $(p{'}_{new}$ - $p{'}_{old})$ value 10,000 times using the same simulation process you used in parts (a) through (g) above.
Store all $(p{'}_{new}$ - $p{'}_{old})$ values in a NumPy array called p_diffs.
new_converted_simulation = np.random.binomial(n_new, p_new, 10000)/n_new
old_converted_simulation = np.random.binomial(n_old, p_old, 10000)/n_old
p_diffs = new_converted_simulation - old_converted_simulation
i. Histogram
Plot a histogram of the p_diffs. Does this plot look like what you expected? Use the matching problem in the classroom to assure you fully understand what was computed here.
plt.hist(p_diffs)
plt.axvline(obs_diff, c='red');
j. What proportion of the p_diffs are greater than the actual difference observed in the df2 data?
print("The proportion of the p_diffs are greater than the actual difference observed in ab_data.csv is : ",(p_diffs > obs_diff).mean())
The proportion of the p_diffs are greater than the actual difference observed in ab_data.csv is : 0.9124
◉ Plotting p_diffs reveals that its have a normally distributed sample distribution. ◉
k. Please explain in words what you have just computed in part j above.
l. Using Built-in Methods for Hypothesis Testing
We could also use a built-in to achieve similar results. Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance.
# number of conversions with the old_page
convert_old = df2[df2['landing_page'] == "old_page"]['converted'].sum()
# number of conversions with the new_page
convert_new = df2[df2['landing_page'] == "new_page"]['converted'].sum()
# number of individuals who were shown the old_page
n_old = len(df2.query('landing_page == "old_page"'))
# number of individuals who received new_page
n_new = len(df2.query('landing_page == "new_page"'))
print("Number of conversions with the old page is : ", convert_old)
print("Number of conversions with the new page is : ", convert_new)
print("Number of individuals with the old page is : ", n_old)
print("Number of individuals with the new page is : ", n_new)
Number of conversions with the old page is : 17489 Number of conversions with the new page is : 17264 Number of individuals with the old page is : 145274 Number of individuals with the new page is : 145310
m. Now use sm.stats.proportions_ztest() to compute your test statistic and p-value. Here is a helpful link on using the built in.
import statsmodels.api as sm
z_score, p_value = sm.stats.proportions_ztest([convert_new, convert_old], [n_new, n_old] , alternative='larger')
print("The Z test score is : ", z_score)
print("The P-value is : ", p_value)
The Z test score is : -1.3109241984234394 The P-value is : 0.9050583127590245
n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages? Do they agree with the findings in parts j. and k.?
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆
In this final part, We will see that the result I acheived in the previous A/B test can also be acheived by performing regression.
a. Since each row in the df2 data is either a conversion or no conversion, what type of regression should you be performing in this case?
b. The goal is to use statsmodels library to fit the regression model you specified in part a. above to see if there is a significant difference in conversion based on the page-type a customer receives. However, you first need to create the following two columns in the df2 dataframe:
intercept - It should be 1 in the entire column. ab_page - It's a dummy variable column, having a value 1 when an individual receives the treatment, otherwise 0. df2['ab_page'] = pd.get_dummies(df['group'])['treatment']
df2['New page'] = pd.get_dummies(df['landing_page'])['new_page']
df2['intercept'] = 1
c. Use statsmodels to instantiate your regression model on the two columns you created in part (b). above, then fit the model to predict whether or not an individual converts.
log_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']])
results = log_mod.fit()
Optimization terminated successfully.
Current function value: 0.366118
Iterations 6
d. Provide the summary of your model below, and use it as necessary to answer the following questions.
results.summary()
| Dep. Variable: | converted | No. Observations: | 290584 |
|---|---|---|---|
| Model: | Logit | Df Residuals: | 290582 |
| Method: | MLE | Df Model: | 1 |
| Date: | Wed, 17 Aug 2022 | Pseudo R-squ.: | 8.077e-06 |
| Time: | 20:47:51 | Log-Likelihood: | -1.0639e+05 |
| converged: | True | LL-Null: | -1.0639e+05 |
| Covariance Type: | nonrobust | LLR p-value: | 0.1899 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| intercept | -1.9888 | 0.008 | -246.669 | 0.000 | -2.005 | -1.973 |
| ab_page | -0.0150 | 0.011 | -1.311 | 0.190 | -0.037 | 0.007 |
e. What is the p-value associated with ab_page?
Why does it differ from the value you found in Part II?
f. Now, you are considering other things that might influence whether or not an individual converts. Discuss why it is a good idea to consider other factors to add into your regression model. Are there any disadvantages to adding additional terms into your regression model?
g. Adding countries
Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in.
You will need to read in the countries.csv dataset and merge together your df2 datasets on the appropriate rows. You call the resulting dataframe df_merged. Here are the docs for joining tables.
Does it appear that country had an impact on conversion? To answer this question, consider the three unique values, ['UK', 'US', 'CA'], in the country column. Create dummy variables for these country columns.
Provide the statistical output as well as a written response to answer this question.
# Read the countries.csv
countries_df = pd.read_csv('./countries.csv')
# Join with the df2 dataframe
df_new = countries_df.set_index('user_id').join(df2.set_index('user_id'), how='inner')
### Creating the necessary dummy variables
countries = df_new.country.unique()
df_new[countries] = pd.get_dummies(df_new.country)[countries]
### Fitting my Logistic Model And Obtaining the Results
log_mod = sm.Logit(df_new['converted'], df_new[['intercept', 'ab_page', 'UK', 'US']])
results = log_mod.fit()
results.summary()
Optimization terminated successfully.
Current function value: 0.366113
Iterations 6
| Dep. Variable: | converted | No. Observations: | 290584 |
|---|---|---|---|
| Model: | Logit | Df Residuals: | 290580 |
| Method: | MLE | Df Model: | 3 |
| Date: | Wed, 17 Aug 2022 | Pseudo R-squ.: | 2.323e-05 |
| Time: | 20:47:53 | Log-Likelihood: | -1.0639e+05 |
| converged: | True | LL-Null: | -1.0639e+05 |
| Covariance Type: | nonrobust | LLR p-value: | 0.1760 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| intercept | -2.0300 | 0.027 | -76.249 | 0.000 | -2.082 | -1.978 |
| ab_page | -0.0149 | 0.011 | -1.307 | 0.191 | -0.037 | 0.007 |
| UK | 0.0506 | 0.028 | 1.784 | 0.074 | -0.005 | 0.106 |
| US | 0.0408 | 0.027 | 1.516 | 0.130 | -0.012 | 0.093 |
h. Fit your model and obtain the results
Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if are there significant effects on conversion. Create the necessary additional columns, and fit the new model.
Provide the summary results (statistical output), and your conclusions (written response) based on the results.
df_new['ab_UK'] = df_new['ab_page'] * df_new['UK']
df_new['ab_CA'] = df_new['ab_page'] * df_new['CA']
logit_mod = sm.Logit(df_new['converted'], df_new[['intercept','ab_page','UK','CA','ab_UK', 'ab_CA']])
result = logit_mod.fit()
result.summary()
Optimization terminated successfully.
Current function value: 0.366109
Iterations 6
| Dep. Variable: | converted | No. Observations: | 290584 |
|---|---|---|---|
| Model: | Logit | Df Residuals: | 290578 |
| Method: | MLE | Df Model: | 5 |
| Date: | Wed, 17 Aug 2022 | Pseudo R-squ.: | 3.482e-05 |
| Time: | 20:47:56 | Log-Likelihood: | -1.0639e+05 |
| converged: | True | LL-Null: | -1.0639e+05 |
| Covariance Type: | nonrobust | LLR p-value: | 0.1920 |
| coef | std err | z | P>|z| | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| intercept | -1.9865 | 0.010 | -206.344 | 0.000 | -2.005 | -1.968 |
| ab_page | -0.0206 | 0.014 | -1.505 | 0.132 | -0.047 | 0.006 |
| UK | -0.0057 | 0.019 | -0.306 | 0.760 | -0.043 | 0.031 |
| CA | -0.0175 | 0.038 | -0.465 | 0.642 | -0.091 | 0.056 |
| ab_UK | 0.0314 | 0.027 | 1.181 | 0.238 | -0.021 | 0.084 |
| ab_CA | -0.0469 | 0.054 | -0.872 | 0.383 | -0.152 | 0.059 |
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆
df2["Date"] = df2.timestamp.dt.date
new_page = df2.query('landing_page == "new_page"').groupby('Date')["converted"].sum().to_frame().reset_index()
old_page = df2.query('landing_page == "old_page"').groupby('Date')["converted"].sum().to_frame().reset_index()
new_page.head()
| Date | converted | |
|---|---|---|
| 0 | 2017-01-02 | 342 |
| 1 | 2017-01-03 | 753 |
| 2 | 2017-01-04 | 763 |
| 3 | 2017-01-05 | 748 |
| 4 | 2017-01-06 | 833 |
old_page.head()
| Date | converted | |
|---|---|---|
| 0 | 2017-01-02 | 359 |
| 1 | 2017-01-03 | 750 |
| 2 | 2017-01-04 | 802 |
| 3 | 2017-01-05 | 792 |
| 4 | 2017-01-06 | 762 |
import plotly.graph_objects as go
fig = go.Figure()
fig.add_trace(go.Scatter(x=new_page.Date, y=new_page.converted, name="New Page",mode = 'lines+markers'))
fig.add_trace(go.Scatter(x=old_page.Date, y=old_page.converted, name="Old Page", mode="lines+markers"))
fig.update_layout(
title="Trending Chart",
xaxis_title="Date",
yaxis_title="Conversions",
legend_title="Legends",
font=dict(
family="Ariel",
size=15,
color="RebeccaPurple"
)
)
fig.show()
fig,ax = plt.subplots(figsize=(15,10),dpi=70)
splot = sns.countplot(x="group", data=df2, palette="viridis")
for p in splot.patches:
if p.get_height() > 0: # -- > i used if statments bcz when i try to use annotators , there is some countries that have negative values , so the position of the value gets in the bar..
splot.annotate(format(p.get_height(), '.1f'),
(p.get_x() + p.get_width() / 2., p.get_height()),
ha = 'center', va = 'center',
size=15,
xytext = (0, 9),
textcoords = 'offset points')
for patch in ax.patches :
current_width = patch.get_width()
diff = current_width - 0.1
# we change the bar width
patch.set_width(.1)
# we recenter the bar
patch.set_x(patch.get_x() + diff * .5)
title_size = 18
ax.set_ylabel("Count",fontsize=title_size)
ax.set_xlabel("Group",fontsize=title_size)
ax.tick_params(labelsize=16,length=0)
plt.tight_layout()
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆
◍ There is no evidence that the new page would increase the e-commerce company's conversion rate, according to the data we calculated in the A/B test and logistic regression (P-values are greater than our designated Type I error level of $\alpha$ = 0.05) , Maybe because this test performed only in 22 days and if the duration was longer it might would have affect our testing results.
◍ I would advise the business to continue using the old page while considering improving their website from time to time that would favourably affect the e-commerce company's conversion rate, and then another A/B test could be performed to determine if these improvements increased the conversion rate or not.
◆▰▰▰▰▰▰▰▰►►►►►►►►►►►►►▰▰▰►►►►►►►►►►►►▰▰▰▰►►►►►►►►►▰▰▰▰▰▰▰►►►►►►►◆